Pack dimension values in time-series aggregation #136216

dnhatn · 2025-10-08T18:01:09Z

With this change, we pack the dimension into a single value before the second aggregation in time-series queries and unpack it afterward. This avoids generating permutations for multi-valued dimensions in the second aggregation, which is not desirable.

For example, the query

TS k8s | STATS max(rate(request)) BY host, tbucket(1minute)

is rewritten as:

TS k8s
 | STATS rate=rate(request), host=VALUES(host) BY _tsid, tbucket=TBUCKET(1minute)
 | EVAL packed_host=PACK_DIMENSION(host)
 | STATS sum(rate) BY packed_host, tbucket
 | EVAL host=UNPACK_DIMENSION(packed_host)
 | KEEP rate, host, tbucket

There is some overhead with packing and unpacking values, but we tried to isolate this behavior to time-series queries with dimension fields only. That is why I chose this approach.

dnhatn · 2025-10-10T04:33:48Z

...ck/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/X-ConstantVector.java.st

 $if(BytesRef)$
-    public BytesRef getBytesRef(int position, BytesRef ignore) {
+    public BytesRef getBytesRef(int position, BytesRef scratch) {
+        scratch.bytes = value.bytes;


I will open a separate PR for this fix.

dnhatn · 2025-10-10T04:34:29Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

        assertThat(Expressions.attribute(rate.field()).name(), equalTo("network.total_bytes_in"));
        LastOverTime lastSum = as(Alias.unwrap(aggsByTsid.aggregates().get(1)), LastOverTime.class);
        assertThat(Expressions.attribute(lastSum.field()).name(), equalTo("network.cost"));
-        Values clusterValues = as(Alias.unwrap(aggsByTsid.aggregates().get(3)), Values.class);


I will update these tests.

dnhatn · 2025-10-10T04:35:26Z

...ain/java/org/elasticsearch/xpack/esql/expression/function/scalar/internal/InternalPacks.java

+        return Math.max(INITIAL_SIZE_IN_BYTES, positionCount);
+    }
+
+    static BytesRefBlock packBytesValues(DriverContext driverContext, BytesRefBlock raw) {


This is where we encode multi-valued block into a single value.

dnhatn · 2025-10-10T04:36:11Z

...ain/java/org/elasticsearch/xpack/esql/expression/function/scalar/internal/InternalPacks.java

+        }
+    }
+
+    static BytesRefBlock unpackBytesValues(DriverContext driverContext, BytesRefBlock encoded) {


And decode here.

dnhatn · 2025-10-10T04:37:57Z

I will also verify this change with the competitive benchmark.

dnhatn · 2025-10-10T04:42:57Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries-rate.csv-spec

+7.203958127639015    | prod            | [eu, us]       | 2024-05-10T00:10:00.000Z
+6.34494062999877     | staging         | us             | 2024-05-10T00:10:00.000Z
+5.700488689624205    | prod            | [eu, us]       | 2024-05-10T00:20:00.000Z
+5.4539153439153445   | prod            | [eu, us]       | 2024-05-10T00:00:00.000Z


We return arrays for grouping dimensions.

kkrik-es · 2025-10-10T07:15:08Z

.../java/org/elasticsearch/xpack/esql/optimizer/rules/logical/TranslateTimeSeriesAggregate.java

-                firstPassAggs.add(newFinalGroup);
+                var valuesAgg = new Alias(g.source(), g.name(), new Values(g.source(), g));
+                firstPassAggs.add(valuesAgg);
+                if (g.isDimension()) {


What happens with labels, i.e. grouping attributes that are neither dimensions nor metrics? I think multi-values were not allowed in dimensions for a long time, so this may be the case for some older configs.

Ok, so we want to pack not only dimensions but also labels. If that's the case, we might need a different approach.

kkrik-es

Looks good, thanks Nhat. Let's check the numbers on competitive benchmark, in case there are big surprises.

…sions

elasticsearchmachine · 2025-10-13T23:36:16Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

dnhatn · 2025-10-13T23:36:31Z

Thanks Kostas!

elasticsearchmachine · 2025-10-13T23:37:45Z

💔 Backport failed

Status	Branch	Result
❌	9.2	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 136216

kkrik-es · 2025-10-14T06:19:53Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Questions ?

Please refer to the Backport tool documentation

With this change, we pack the dimension into a single value before the second aggregation in time-series queries and unpack it afterward. This avoids generating permutations for multi-valued dimensions in the second aggregation, which is not desirable. For example, the query ``` TS k8s | STATS max(rate(request)) BY host, tbucket(1minute) ``` is rewritten as: ```esql TS k8s | STATS rate=rate(request), host=VALUES(host) BY _tsid, tbucket=TBUCKET(1minute) | EVAL packed_host=PACK_DIMENSION(host) | STATS sum(rate) BY packed_host, tbucket | EVAL host=UNPACK_DIMENSION(packed_host) | KEEP rate, host, tbucket ``` There is some overhead with packing and unpacking values, but we tried to isolate this behavior to time-series queries with dimension fields only. That is why I chose this approach. (cherry picked from commit c8dc04d) # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

With this change, we pack the dimension into a single value before the second aggregation in time-series queries and unpack it afterward. This avoids generating permutations for multi-valued dimensions in the second aggregation, which is not desirable. For example, the query ``` TS k8s | STATS max(rate(request)) BY host, tbucket(1minute) ``` is rewritten as: ```esql TS k8s | STATS rate=rate(request), host=VALUES(host) BY _tsid, tbucket=TBUCKET(1minute) | EVAL packed_host=PACK_DIMENSION(host) | STATS sum(rate) BY packed_host, tbucket | EVAL host=UNPACK_DIMENSION(packed_host) | KEEP rate, host, tbucket ``` There is some overhead with packing and unpacking values, but we tried to isolate this behavior to time-series queries with dimension fields only. That is why I chose this approach. (cherry picked from commit c8dc04d) # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java Co-authored-by: Nhat Nguyen <[email protected]>

elasticsearchmachine added the v9.3.0 label Oct 8, 2025

dnhatn force-pushed the pack-dimensions branch 3 times, most recently from 49bf9c8 to 25b5f7e Compare October 10, 2025 03:51

Pack dimension values in time-series aggregation

d138266

dnhatn commented Oct 10, 2025

View reviewed changes

dnhatn added v9.2.0 >non-issue labels Oct 10, 2025

dnhatn requested review from kkrik-es and martijnvg October 10, 2025 04:36

dnhatn force-pushed the pack-dimensions branch from c6a108c to d138266 Compare October 10, 2025 04:41

dnhatn commented Oct 10, 2025

View reviewed changes

[CI] Auto commit changes from spotless

f437a40

kkrik-es reviewed Oct 10, 2025

View reviewed changes

kkrik-es approved these changes Oct 10, 2025

View reviewed changes

dnhatn added 8 commits October 11, 2025 22:24

all labels

525ffbe

Merge remote-tracking branch 'elastic/main' into pack-dimensions

e347009

Merge remote-tracking branch 'dnhatn/pack-dimensions' into pack-dimen…

7574caa

…sions

Fix tests

4cd2a07

Merge remote-tracking branch 'elastic/main' into pack-dimensions

731be5b

Fix tests

17e9a69

Merge remote-tracking branch 'elastic/main' into pack-dimensions

734b677

Merge remote-tracking branch 'elastic/main' into pack-dimensions

ba22060

dnhatn added :StorageEngine/TSDB You know, for Metrics auto-backport Automatically create backport pull requests when merged labels Oct 13, 2025

dnhatn marked this pull request as ready for review October 13, 2025 23:35

elasticsearchmachine added the Team:StorageEngine label Oct 13, 2025

dnhatn merged commit c8dc04d into elastic:main Oct 13, 2025
34 checks passed

dnhatn deleted the pack-dimensions branch October 13, 2025 23:36

elasticsearchmachine added the backport pending label Oct 13, 2025

kkrik-es mentioned this pull request Oct 14, 2025

[9.2] Pack dimension values in time-series aggregation (#136216) #136519

Merged

Pack dimension values in time-series aggregation #136216

Pack dimension values in time-series aggregation #136216

Uh oh!

Conversation

dnhatn commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dnhatn Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

dnhatn Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

dnhatn Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

dnhatn Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

dnhatn commented Oct 10, 2025

Uh oh!

dnhatn Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

kkrik-es Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

dnhatn Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

kkrik-es left a comment

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 13, 2025

Uh oh!

dnhatn commented Oct 13, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 13, 2025

💔 Backport failed

Uh oh!

kkrik-es commented Oct 14, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dnhatn commented Oct 8, 2025 •

edited

Loading